Skip to content

feat: accept SQL_QUERY dashboard filters in the API, external API, and MCP - #3674

Open
karl-power wants to merge 2 commits into
sql-dashboard-variablesfrom
sql-dashboard-variables-api
Open

karl-power wants to merge 2 commits into
sql-dashboard-variablesfrom
sql-dashboard-variables-api

Conversation

@karl-power

Copy link
Copy Markdown
Contributor

Summary

Lets the API, external API v2, and MCP save and return SQL_QUERY dashboard filters, and checks their connection on save. This is the second of four PRs; the dropdown still has no values and the UI can't create the filter yet.

  • The internal dashboard route rejects a SQL_QUERY filter whose connection doesn't belong to the team. A connection the stored filter with the same ID already used is skipped, matching how filter sources are checked.
  • The external API and MCP expose the filter with a connectionId field, translated to and from the internal connection. The PR 1 stopgap that hid the type is removed; ADHOC stays hidden.
  • The external API's connection check now includes filter connections, so an unknown one fails with "Could not find the following connection IDs". The source check only reads filters that have a sourceId.
  • MCP returns non-blocking warnings when the filter's SQL uses $__sourceTable or $__filters (the filter has no source), leaves a macro unclosed, or references an unknown variable.
  • MCP tool descriptions and the dashboard prompt describe the filter, including $__filter(<column>, $<variable>) for reading another filter's selection. openapi.json is regenerated.

How to reproduce

Starting point: yarn dev, with the MCP server connected.

On the base branch (#3672):

  1. Call clickstack_save_dashboard with a filter { "type": "SQL_QUERY", "name": "Service", "connectionId": "<connection ID>", "sqlTemplate": "SELECT DISTINCT ServiceName FROM default.otel_logs WHERE $__timeFilter(Timestamp) ORDER BY 1", "variableName": "svc" }.
  2. The call fails schema validation, since the type isn't exposed.

On this branch:

  1. Repeat the call. The dashboard saves and the response includes the filter with connectionId.
  2. Add a raw-SQL tile containing $__filter(ServiceName, $svc) and run clickstack_query_tile with and without variableValues: [{ "name": "svc", "values": ["<service>"] }]. The tile narrows to that service only when a value is selected.
  3. Save with connectionId: "000000000000000000000000". The call fails with "Could not find the following connection IDs".
  4. Save with sqlTemplate: "SELECT 1 FROM $__sourceTable". The dashboard saves and returns a warning that the filter has no source.

How to test on Vercel preview

N/A — non-UI change

References

@karl-power
karl-power requested a review from a team as a code owner November 16, 2026 12:09
@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 16, 2026 3:53pm UTC
hyperdx-storybook Ready Ready Preview Nov 16, 2026 3:53pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: f049f92

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@karl-power
karl-power added this pull request to stack #3675 November 16, 2026 12:10
@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

1 new clone introduced by this PR.

  • packages/common-utils/src/core/utils.ts:988-1008 ↔ packages/common-utils/src/core/utils.ts:1036-1056 — 21 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.18%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

2 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 2 minor

1 posted as inline comment(s) on the changed lines. 1 listed below.

Findings outside the changed lines

1 minor
  • 🔵 packages/api/src/mcp/tools/sources/listSources.ts:369 — list_sources still tells the agent that connectionId is only needed for clickstack_sql → The new SQL_QUERY filter descriptions (schemas.ts mcpSqlQueryFilterSchema.connectionId, prompts/dashboards/content.ts) tell the agent to take connectionId from clickstack_list_sources. But that tool's nextStep still says 'connectionId is only needed for clickstack_sql.', which contradicts them. Update the hint to also mention SQL_QUERY filters (and raw SQL tiles), or drop the 'only'.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

The change adds SQL_QUERY dashboard filters to the internal API, external API v2, and MCP, with a save-time team-ownership check on the filter's ClickHouse connection and symmetric connection ↔ connectionId translation. I verified the team-isolation path end to end: validateFilterConnections (packages/api/src/routers/api/dashboards.ts) mirrors the existing validateFilterSources skip-existing semantics, existingFilters is sourced from the already-team-scoped stored dashboard, and getConnectionsByTeam(teamId) scopes the lookup — no cross-team bypass is reachable, since no SQL_QUERY filter could have been persisted before this PR. The MCP and external paths re-parse through externalDashboardFilterSchema, which supplies the required isBroadcastEnabled:false / isVariableEnabled:true literals, so persisted internal filters stay schema-valid. Prior review feedback (connection-swap PATCH test, validateSqlQueryFilterTemplate wiring, the 'sourceId' in filter decision) is addressed in the diff.

✅ No critical issues found.

🟡 P2 -- recommended

  • packages/api/src/routers/external-api/__tests__/dashboards.int.test.ts:848 -- The external-API SQL_QUERY suite covers create and GET round-trip but never exercises an update that introduces or swaps a filter connectionId, so the external re-validation path for filter connections is untested even though the internal route has explicit PATCH coverage.
    • Fix: Add an external-API update test that submits a SQL_QUERY filter with a connection outside the team and asserts a 400, plus one that moves it to another team-owned connection and asserts success.
🔵 P3 nitpicks (2)
  • packages/api/src/routers/api/dashboards.ts:124 -- validateFilterConnections and the external getMissingConnections (packages/api/src/routers/external-api/v2/utils/dashboards.ts:1110) implement the same team-ownership check with divergent shapes and messages, a drift risk as filter-connection rules evolve.
    • Fix: Factor the connection-membership check into one shared helper reused by both routes, or track the divergence as a follow-up.
  • packages/api/src/mcp/tools/dashboards/validation.ts:470 -- getFilterVariableWarnings overloads the local where variable to also hold a sqlTemplate, which obscures that two structurally different filter inputs flow through the same validation branch.
    • Fix: Rename the variable to reflect that it is the dropdown-values query regardless of source field, or branch the SQL_QUERY case explicitly.

Reviewers (9): correctness, security, adversarial, api-contract, kieran-typescript, testing, maintainability, previous-comments, project-standards.

Testing gaps: External-API update path for SQL_QUERY filter connections is unverified; internal PATCH create/move/delete-tolerance cases are covered.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 468 passed • 1 skipped • 1069s

Status Count
✅ Passed 468
❌ Failed 0
⚠️ Flaky 0
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@karl-power

Copy link
Copy Markdown
Contributor Author

Review follow-up:

  • Deep Review P2 (connection swap test): added in e5c0930. PATCH to another team-owned connection returns 200.
  • Deep Review P3 ('sourceId' in filter): keeping it. The old !isStaticListFilter check assumed every other filter type has a source, which is what broke when SQL_QUERY was added. The property check doesn't break that way for new types.
  • clickstack_list_sources nextStep text: valid, but it was already wrong for raw SQL tiles and this PR doesn't touch that file. Will fix in a follow-up.
  • Duplicate-code report (common-utils/src/core/utils.ts): not part of this PR's diff against its base (feat: add SQL_QUERY dashboard filter type to the schema #3672); it shows up because the check compares against main.

@github-actions github-actions Bot added the review/tier-4 Critical — deep review + domain expert sign-off label Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Critical-path files (2) — tenancy, public API, or shipped database config:
    • packages/api/src/routers/external-api/v2/dashboards.ts
    • packages/api/src/routers/external-api/v2/utils/dashboards.ts

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 9
  • Production lines changed: 376 (+ 237 in test files, excluded from tier calculation)
  • Critical-path lines changed: 97
  • Branch: sql-dashboard-variables-api
  • Author: karl-power

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

…d MCP

Validate a SQL query filter's connection on save, and let the external API
and MCP read and write the filter.

- Reject dashboard saves whose SQL_QUERY filter names a connection outside
  the team, skipping a connection the stored filter already used
- Map the filter's connection to connectionId in the external API and MCP,
  and stop hiding the type there
- Check filter connections in the external API's connection validation
- Warn in MCP about source-only macros, unclosed macros, and unknown
  variables in the filter's SQL
- Document the filter in the MCP tool descriptions, prompts, and OpenAPI spec

Refs HDX-5524
@karl-power
karl-power force-pushed the sql-dashboard-variables-api branch from e5c0930 to f049f92 Compare November 16, 2026 15:50
* `validateFilterSources`, a connection the stored filter with the same ID
* already used is skipped. Returns an error message, or null.
*/
async function validateFilterConnections(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — validateFilterConnections reimplements the team-connection lookup in getMissingConnections

Both functions collect connection IDs, call getConnectionsByTeam, build a set of _id.toString() and return the IDs missing from it. The existing one is getMissingConnections in packages/api/src/routers/external-api/v2/utils/dashboards.ts:1589. Move one helper (for example getMissingTeamConnectionIds(teamId, ids)) into controllers/connection.ts and call it from both places, so the ownership rule has a single source of truth.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

This branch was successfully deployed

2 active deployments
Preview – hyperdx-storybook — f049f922 Deployed Nov 16, 2026 by vercel[bot]
Preview – hyperdx-oss — f049f922 Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant